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2 0 Three-dimensional model processing method, and apparatus therefor. 
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0 A three-dimensional model processing apparatus includes a three-dimensional data-format converter for 
converting the data format of three-dimensional data inputted by a three-dimensional data input unit, a three- 
dimensional data processor for processing the three-dimensional data converted by the three-dimensional data- 
format converter, a three-dimensional model creating unit for creating a three-dimensional model from the three- 
dimensional data processed by the three-dimensional processor, and a three-dimensional model editor for 
editing the three-dimensional model created by the three-dimensional model creating unit and applying 
operations to the shape data. A three-dimensional model is created and edited irrespective of the type of three- 
dimensional data inputted in different modes. 
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BACKGROUND OF THE INVENTION 

1. Field of the Invention: 

This invention relates to a three-dimensional model .processing method and apparatus for forming a 
three-dimensional model of an object, editing the model, adding shading or the like and displaying the 
model, without relying upon the data format of inputted three-dimensional data. 

2. Description of the Related Art: 

Various three-dimensional model processing apparatus have been disclosed heretofore. In one example 
of such an apparatus, three-dimensional position data indicative of the surface of an actual object such as a 
clay model are entered by a contacting-type three-dimensional coordinate measuring unit such as a three- 
dimensional digitizer, and a three-dimensional model is created from the entered three-dimensional data. In 
another example, the aforementioned three-dimensional position data are entered by a space-sectioning 
type three-dimensional measuring unit and a three-dimensional model is created from these three- 
dimensional data. 

A three-dimensional model processing apparatus of this kind makes it possible to create a three- 
dimensional model from three-dimensional data. 

In accordance with the conventional three-dimensional model processing apparatus, however, the three- 
dimensional data are entered in a data format that is dependent upon the three-dimensional measuring unit, 
and therefore a three-dimensional model cannot be created by processing three-dimensional data having a 
different data format. In addition, once a three-dimensional model has been created, the three-dimensional 
data cannot be converted into three-dimensional data having another format- 
Several specific examples will now be described. 

A range-image processing method, in which a range image is the object of processing and information 
indicating the characteristics of the surface shape of a three-dimensional object in the image is detected, is 
described in "A Hybrid Approach to Range Image Segmentation Based on Differential Geometry" by 
Naokazu Yokoya and Martin D. Levine, Joho Shori Gakkai Ronbunshi, Vol. 30. No. 8, pp. 944 - 953 
(1989). 

In this conventional technique for processing range images, however, the relationship between the 
coordinate system in which an object is placed and the coordinate system of the image is not defined, and 
it is necessary to give parameters such as the aspect ratio along each of the coordinate axes each time 
processing is performed. 

Various types of non-contacting arrangements have been attempted. For example, there is a method of 
measuring the time of flight of light or a method in which distance data are inputted by a three-dimensional 
measuring unit such as a range finder based upon triangulation. and a three-dimensional model is created 
from the three-dimensional data obtained. 

In a three-dimensional model creating apparatus which utilizes such a three-dimensional measuring unit, 
an input sensor is directed with respect to the object to be entered, and points not concealed by other 
objects can be inputted accurately. However, a drawback is that distance data regarding portions in back of 
the input sensor and points that are concealed cannot be inputted. 

In an effort to eliminate this drawback, a three-dimensional model is obtained by either of the following 
methods in the prior art: (1) The sensor is fixed and the object is revolved about a main axis, in 
synchronization with which a range image is measured. (2) The sensor is revolved about the object, in 
synchronization with which a range image is measured. 

However, both of these methods require a mechanism for revolving the object or the sensor. In addition, 
it is not easy to reconstruct a three-dimensional shape by judging corresponding pixels of a plurality of 
range images. 

In general, when an image in which an object in three-dimensional space is projected into two- 
dimensional space is outputted, shading is applied. Then, when the image is displayed, for example, 
images in which the object is seen from a plurality of directions are displayed in continuous fashion, 
whereby the orientation of the object surface is outputted so that the object in three-dimensional space is 
expressed in two-dimensional space. 

With this conventional method, however, the entirety of the surface shape of the object in a certain 
three-dimensional space cannot always be outputted in the form of a single two-dimensional image. 

In another example of a conventional three-dimensional model processing apparatus, it has been 
attempted to input three-dimensional coordinate data indicative of the surface of an object by a three- 
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dimensional coordinate measuring unit such as a laser range finder, and create a three-dimensional model 
by triangular patches in which all of the three-dimensional coordinate data serve as the vertices. Alter- 
natively, it has been attempted to create a three-dimensional model based upon triangular patches in which 
some of the original data are adopted as the vertices, as by widening the spacing between neighboring 

5 points of the three-dimensional coordinates serving as the vertices. A three-dimensional model is capable of 
being created from three-dimensional data in a three-dimensional model processing apparatus of this kind. 

Further, in another example of a conventional three-dimensional model processing apparatus, it has 
been attempted to input three-dimensional data, which are indicative of the surface of an actual object such 
as a clay model, by a contacting-type three-dimensional measuring unit such as a three-dimensional 

w digitizer, and create a three-dimensional model from these three-dimensional data. Further, it has been 
attempted to enter the three-dimensional position data by a space-sectioning type three-dimensional 
measuring unit and create a three-dimensional model from these three-dimensional data. 

In addition, attempts have been to utilize the obtained three-dimensional objects in various animation or 
CAD/CAM techniques. 

/5 With the conventional three-dimensional model processing apparatus, however, the three-dimensional 
model is created by selecting representative points uniformly over the entire area of the object surface from 
the three-dimensional coordinate data of the object surface inputted by the three-dimensional coordinate 
measuring unit such as the aforementioned laser range finder, and creating the model based upon 
polygonal patches in which the uniform representative points serve as the vertices. A problem which arises 

20 is that the amount of data representing the model becomes extremely large when it is attempted to raise 
accuracy. 

The amount of data can be reduced somewhat in the case of the three-dimensional mode! created by 
the triangular patches in which some of the data serve as the vertices, wherein the spacing between 
neighboring points of the three-dimensional coordinates serving as the vertices is widened. However, at 

25 complicated portions or curved surfaces where the shape is highly irregular, the surface of the object 
cannot be described correctly. On the other hand, the vertices of polygonal patches are placed at intervals 
equal to those of other portions even at large planar areas, and therefore unneeded data also is present. 

Further, in order to create 3-D animation video in the prior art. this is possible only when a specialist or 
a CG animator utilizes an expensive system. An amateur encounters great difficulty when describing the 

30 modeling and moving of 3-D shapes. These systems can be utilized by relying upon manual operations or 
the artistic expertise of craftsmen, and many operations, such as modeling and the designation of motion, 
cannot be performed automatically. 

SUMMARY OF THE INVENTION 

35 

An object of the present invention is to make possible the creation of three-dimensional models, the 
editing thereof and the display of such models with added shadows, irrespective of the kind of three- 
dimensional data entered in different formats. 

Another object of the present invention is to make it possible to convert the data of a three-dimensional 
40 model that has been created into a three-dimensional model having another format. 

By way of example, by providing a step of converting the type of a range image and normalizing each 
axis, pixel values are calculated in conformity with the type of data, which is given for each pixel of an 
inputted range image, using parameters such as aspect ratio along each axis of the coordinate system. 

A further object of the present invention is to provide a three-dimensional model creating method and 
45 apparatus in which it is possible to provide shape data of an unknown portion on the basis of a range image 
obtained from one direction. 

More specifically, with regard to an object having axial symmetry, the user designates the axis of 
symmetry interactively with respect to an image whose distance is measured from one direction, and 
already existing position information is utilized to interpolate data, thereby constructing a three-dimensional 
50 model to obtain a three-dimensional model over the entire surface of the object. 

Still another object of the present invention is to provide an image output method and apparatus in 
which the entirety of the surface shape of an object in three-dimensional space is expressed by an output 
of a single two-dimensional image from a certain direction. 

More specifically, the direction of the object surface at each point is calculated from shape data 
55 inputted from a three-dimensional shape input unit, a three-dimensional data creating unit or a three- 
dimensional shape data base, and information possessed by surface-direction data or the method of 
symbolization is displayed while being selected in dialog fashion in conformity with the user's objectives. 

A further object of the present invention is to make it possible to create a three-dimensional model in 

4 
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which the shape of the model is correctly represented with a small amount of data, this being accomplished 
by selecting representative points that are closely spaced on complex portions or curved surfaces where 
shape is highly irregular, selecting representative points that are distantly spaced on large flat surfaces, and 
adopting these representative points as the vertices of polygonal patches. 

5 As a result, it is possible to create a three-dimensional model, wherein the shape of the, model is 
correctly represented with a small amount of data, by selecting representative points that are closely 
spaced on complex portions or curved surfaces where shape is highly irregular, selecting representative 
points that are distantly spaced on large flat surfaces, and adopting these representative points as the 
vertices of polygonal patches. 

10 Further, after an inputted range image has been converted into geometric data indicative of a mesh 
shape, a polygon or a freely curved surface in conformity with the characteristics of an input object, various 
transformation operations (designating, bending and twisting the skeleton of a transformed curved surface) 
are performed selectively and the transformation operations are interpolated to create animation video. As a 
result, the user need only designate how an object is to be transformed and animated without having a 

;5 detailed knowledge of modeling. This arrangement can be utilized by anyone, without requiring special skill, 
and therefore provides a system exhibiting a high productivity. 

Other features and advantages of the present invention will be apparent from the following description 
taken in conjunction with the accompanying drawings, in which like reference characters designate the 
same or similar parts throughout the figures thereof. 

20 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a block diagram showing the construction of a three-dimensional model processing apparatus 
according to an embodiment of the present invention; 
25 Fig. 2 is a block diagram showing the arrangement of hardware constituting the three-dimensional model 
processing apparatus according to this embodiment; 

Fig. 3 is a diagram showing the construction of a memory according to a first embodiment; 
Fig. 4 is a flowchart showing the overall processing of the first embodiment; 
Fig. 5 is a diagram showing an example of a window according to the first embodiment; 
30 Fig. 6 is a diagram showing an example of operators for directly obtaining linear partial differentials and 
quadratic partial differentials from coefficients of a quadratic surface subjected to fitting according to the 
first embodiment: 

Fig. 7 is a diagram showing an example of an operator for directly obtaining linear partial differentials 

and quadratic partial differentials from an original range image according to the first embodiment; 
35 Fig. 8 is a diagram for describing estimation of a selective local plane; 

Fig. 9 is a diagram showing an example of types of planes according to the first embodiment; . 

Fig. 10 is a diagram showing the construction of a memory according to a second embodiment; 

Fig. 1 1 is a flowchart showing a basic algorithm according to the second embodiment; 

Fig. 12 is a diagram showing a method of deciding a main axis from drawings of three surfaces 
40 according to the second embodiment; 

Fig. 13 is a flowchart showing an algorithm for the construction of a three-dimensional model according 

to the second embodiment; 

Fig. 14 is a diagram showing an example of the description of a three-dimensional (polygonal) model as 
well as a cubic shape: 
45 Fig. 15 is a diagram showing the scanning state of a measured range image; 

Fig. 16 is a flowchart showing an algorithm for calculating phase information of a ridge line; 

Fig. 17 is a diagram illustrating pseudo-coding of an algorithm for calculating phase information of a 

ridge line; 

Fig. 18 is a diagram showing the construction of a memory according to a third embodiment; 
50 Fig. 19 is a flowchart showing a basic algorithm according to the third embodiment; 

Fig. 20 shows examples in which the sense of the surface of an object surface is displayed as lengths 
and directions of line segments; 

Fig. 21A shows examples in which the sense of the surface of an object surface is displayed as the color 
of each point; 

55 Fig. 21 B shows original color images corresponding to Fig. 21 A; 

Fig. 22 is a flowchart showing an algorithm for converting a unit normal vector into a color; 
Fig. 23 is a diagram showing the construction of a memory according to a fourth embodiment; 
Fig. 24 is a flowchart showing the overall processing of the fourth embodiment; 
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Fig. 25. is a flowchart of surface classification processing according to the fourth ennbodinnent; 
Fig. 26 is a table showing surface classes and the codes thereof; 

Fig. 27 is a flowchart showing a processing procedure for creating a polygonal-patch model according to 
the fourth embodiment; 

5 Fig. 28 is a diagram schematically showing an example of processing of range-image data according to 
the fourth embodiment; 

Fig. 29 is a flowchart showing a processing procedure for contour extraction according to the fourth 
embodiment; 

Fig, 30 is a diagram showing a polygonal-patch data table according to the fourth embodiment; 
w Fig. 31 is an explanatory view for creating a polygonal-patch model of each area; 

Fig. 32 is a diagram showing the construction of a memory according to a fifth embodiment; 
Fig. 33 is a flowchart showing the overall processing of the fifth embodiment; 

Fig. 34 is a flowchart showing a processing procedure for creating a polygonal-patch model according to 

the fifth embodiment; . 
;5 Fig. 35 is a diagram for describing the scanning of a range image; 

Fig. 36 is a diagram for describing a method of selecting characteristic points; 

Fig. 37 is a diagram for describing an example of points on a side to be revised; 

Fig. 38 is a diagram showing an example of data indicative of a polygon model; 

Fig. 39 is a diagram showing the construction of a memory according to a sixth embodiment; 
20 Fig. 40 is a flowchart showing the overall processing of the sixth embodiment; 

Fig. 41 is a diagram showing an example of data indicative of a polygon model; 

Fig. 42 is a diagram for describing an example of transformation operations according to the sixth 
embodiment; 

Fig. 43A is a diagram showing an example of the results of transformation according to the sixth 
25 embodiment; 

Fig. 43B shows an original color image corresponding to Fig. 43A; 

Fig. 44 is a diagram for describing an example of the interpolation of transformation parameters 
according to the sixth embodiment; and 

Fig. 45 is a diagram showing an example of pseudo-coding for creating ridge-line phase information. 

30 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Before describing embodiments of the invention in detail, the technical terms used herein will be simply 
defined. 

35 A "range image" refers to an image in which the vertical distance (parallel projection) between each 
point on the surface of an object measured by a range-measuring sensor and a reference plane is given. As 
for the data format, distance (Z value) at each point of an (X,Y) square two-dimensional lattice is stored. 
These are data which recently have begun to be utilized in the field of computer vision and computer 
graphics. 

40 Next, a "shape model", unlike a -range image, is expressed as geometrical information such as the 
vertices and sides of a 3-D object. For example, there are mesh models, polygon models, quadratic-surface 
models and freely curved-surface models, mentioned in order starting from those which have a low 
description level. This is a data format often utilized in CG and CAD systems. 

A "mesh model" is a triangular lattice model in which the x and y coordinates are spaced apart 

45 equidistantly. As shown in the drawings, 3-D geometric data can be created by connecting the diagonals of 
a lattice shape and converting distance data into a series of triangular polygons. Basically, this model can 
readily be realized by a simple filter program. 

A "polygon model" is composed of arbitrary point sequences and is a polygon in an identical plane. 
The structure of the polygon data is shown in the drawings. The data structure is broadly divided into two 

50 portions. The first half is a portion in which vertex information constituting a polygon is designated. This is 
an (x,y,2) value of vertex information and a normal vector (Nx.Ny.Nz) value at an optional vertex; these 
values are essential data. As for the second half, connection information of ridge lines constituting a polygon 
is indicated with respect to the designated vertex information. There are initially n-number of ridge lines 
constituting a polygon, and these are followed by n-number of index values constituting a polygon. 

55 A "freely curved-surface model" is a model having the highest description capability and is widely used 
in CAD and the like. A Bezier or NURBS (non-rational B-spline surface) curved surface is known as a freely 
curved surface. A detailed description of the technique is found in "An Adaptive Subdivision Method for 
Surface-Fitting from Sampled Data", Francis Schm Computer Graphics, vol. 20, number 4, 1986. 

6 
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Preferred embodiments of the present invention will now be described in detail witfi reference to the 
accompanying drawings. 

Fig. 1 is a block diagram showing the construction of a three-dimensional model processing apparatus 
according to this embodiment. 

5 In Fig. 1. numeral 11 denotes a three-dimensional data-format converter for converting the data format 

of three-dimensional data 12 inputted by a three-dimensional data input unit. Numeral 13 denotes a three- 
dimensional data processor for processing the three-dimensional data converted by the converter 11. A 
three-dimensional data measuring unit 14 performs various measurements with regard to three-dimensional 
data converted by the converter 1 1. A three-dimensional model creating unit 15 creates a three-dimensional 

10 model from the three-dimensional data processed by the three-dimensional processor 13. A three- 
dimensional model editor 16 edits the three-dimensional model created by the creating unit 15 and applies 
operations to the shape data. A three-dimensional model display unit 17 applies shading to the three- 
dimensional model created by the creating unit 15 or edited by the editor 16 and displays the resulting 
model. A three-dimensional model reverse-converter 18 reverse-converts, into three-dimensional data 12* 

;5 having another format, the three-dimensional model created by the creating unit 15 or edited by the editor 
16. 

Fig. 2 is a block diagram showing the basic construction of the hardware constituting the three- 
dimensional model processing apparatus. This hardware is the same for all of the following embodiments, 
including this embodiment. 

20 Shown in Fig. 2 are a program memory 1 for storing various processing procedures, a data memory 2 
which stores information necessary for processing in this system as well as input/output data, a CPU 3 for 
performing various processing and controlling various elements of the apparatus in accordance with the 
processing procedures stored in the program memory 1, and a display unit 4 for displaying 3-D shape 
models obtained by the system, and for interactively displaying instructions from the user. In this 

25 embodiment, the display unit is of the multiple-window type. Numeral 5 denotes a mouse for inputting 
commands from the user, 6 a keyboard (KB) which the user employs to create a program and to input 
commands to the system, and 7 a laser range finder for measuring the vertical distance between each point 
on an object surface and a reference plane, and for inputting a range image. 

An example of the operation of the foregoing arrangement will now be described. 

30 When the three-dimensional data 12 are inputted to the three-dimensional data converter 11, the latter 
converts the data format into a standard data format in conformity with the data format of the three- 
dimensional data 12. 

[Example 1] 

35 

One example of processing will be described with regard to a specific case. 

Assume that a laser beam in which the strength of the laser light has been modulated is emitted from 
the laser range finder, and that the phase difference between the waves reflected from the object and the 
original waves is detected, thereby measuring the time of flight of the light to provide three-dimensional 
40 data. 

The three-dimensional data-format converter 1 1 converts the three-dimensional data into the format of a 
range image, which is one "standard data format" of the present system. More specifically, the converter 11 
makes a conversion from phase difference to distance, transforms the coordinate system and performs 
normalization along each axis. Here the range image refers to an image in which the vertical distances 

45 between visual points and the surface of the object are given as values of pixels. 

The three-dimensional data processor 13 performs segmentation, such as the extraction of planes and 
various curved-surface regions, the extraction of ridge lines, etc., from the range image, and calculates 
attribute values of pixels, curvature, etc. Next, label image processing such as the elimination of very small 
regions, the joining of regions and the growing of regions based upon the attribute values is executed. 

50 On the basis of the results of processing performed by the three-dimensional data processor 13, the 
three-dimensional model creating unit 15 creates a three-dimensional model, such as a polygon model, a 
parametric model, a model based upon boundary representation or a solid model. 

A procedure for creating a polygon model will be described in general terms as one example. A 
polygon having vertices on the borderline of a planar region is fitted to the planar region, which is extracted 

55 by the three-dimensional data processor 13, and a polygon having vertices which are points selected at a 
certain spacing based upon the type of the curved surface and the size of its curvature is fitted to an 
extracted curved-surface region. The coordinates of the vertices of each of the polygons thus created, as 
well as the connectivity of these points, are recorded and a polygon model is created. 
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The three-dimensional model display unit 17 presents a realistic display of the three-dimensional model. 

which h3s been obtained by the three-dimensional model creating unit 15 or the three-dimensional model 

editing onit 16. by providing such parameters as points of view, positions of light sources, reflectance and 

transparency of the object surface as well as color. 
5 The three-dimensional data measuring unit 14 performs various measurements with regard to the 

standard data format created by the three-dimensional data-format converter 1 1 . For example, in case of the 

aforesaid range image, the measuring unit 14 obtains the distance, in three-dimensional space, between two 

arbitrarily given points on the image and computes the area and volume, in three-dimensional space, of any 

closed region drawn on the image. 
w The three-dimensional model reverse-converter 18 converts the data of the three-dimensional nnodel, 

which has been obtained by the three-dimensional model creating unit 15 or the three-dimensional model 

editor 16, into three-dimensional data having another format. 

A method of converting the aforesaid polygon model into a range image will be described as one 

example. The conversion can be made by recording the Z value of each point using a Z buffer algorithm, 
;5 which is one type of processing for eliminating concealed surfaces. 

[Example 2] 

Another example of processing will be described with regard to a specific case. 
20 Assume that several computed tomographs (CT images) obtained by X rays or NMR are available as 
three-dimensional data. 

The three-dimensional data-format converter 1 1 converts the three-dimensional data into the format of a 
three-dimensional image, which is one "standard data format" of the present system. More specifically, the 
converter 1 1 makes a conversion of sampling interval (resolution). The three-dimensional image mentioned 

25 here refers to an image in which three-dimensional space is segmented into cubic lattices and a density 
value is assigned to each small cube. 

The three-dimensional data processor 13 performs segmentation processing, such as the extraction of 
characteristic regions, the extraction of surfaces, the extraction of planes and various curved-surface 
regions, and the extraction of ridge lines, etc., from the three-dimensional image. 

30 On the basis of the results of processing performed by the three-dimensional data processor 13. the 
three-dimensional model creating unit 15 creates a three-dimensional model, such as a polygon model, a 
parametric model, a model based upon boundary representation or a solid model. 

A procedure for creating a solid model will be described in general terms as one example. With regard 
to a characteristic region extracted by the three-dimensional data processor 13, an "occupied" code is 

35 applied to the small cubes in the interior, an "unoccupied" code is applied to small regions which are 
regions other than the above, and a solid model of the object is created as a set of the small cubes filling 
the interior. 

The three-dimensional model editor 16 performs operations such as changing the shape parameters of 
the three-dimensional model' obtained by the three-dimensional model creating unit 15 and combining the 
40 shape models of a plurality of objects that have been stored in advance. 

The three-dimensional model display unit 17 presents a realistic display of the three-dimensional model, 
which has been obtained by the three-dimensional model creating unit 15 or the three-dimensional model 
editing unit 16, by providing such parameters as points of view, positions of light sources, reflectance and 
transparency of the object surface as well as color. The display unit 17 starts display routines conforming to 
45 the types of respective shape models. 

The three-dimensional data measuring unit 14 - performs various measurements with regard to the 
standard data format created by the three-dimensional data-format converter 11. For example, in case of the 
aforesaid three-dimensional image, the measuring unit 14 obtains the distance, in three-dimensional space, 
between two arbitrarily given points in the image and computes the area and volume, in three-dimensional 
50 space, of any closed region drawn in th^^ :mage. 

The three-dimensional model revr;r5-:.r-converter 18 converts the data of the three-dimensional model, 
which has been obtained by the three -..tmensional model creating unit 15 or the three-dimensional model 
editor 16. into three-dimensional data having another format. 

A method of converting the aforesaid solid model into a range image will be described as one example. 
55 The conversion can be made by recording the Z value corresponding to each point using a Z buffer 
algorithm, which is one type of processing for eliminating concealed surfaces. 

In accordance with the three-dimensional model processing apparatus of this embodiment, as described 
above, a three-dimensional model is capable of being created irrespective of the type of three-dimensional 
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data inputted in a different format, and the model can be edited. Furthermore, the model can be displayed 
with shading if required. In addition, once a three-dimensional model has been created, the data thereof can 
be converted into three-dimensional data having a different format. 

5 (Normalization of axes and range image processing) 

Fig. 3 is a diagram showing the programs stored in the program memory 1 and the data stored in the 
data memory 2. The programs stored in the program memory 1 shown in (a) of Fig. 3 areas follows: an axis 
normalizing program 201 for converting the type of range image and normalizing each axis; a jump-edge 

70 detecting program 202 for detecting jump edges from a range image; a linear partial-differential calculating 
program 203 for calculating a linear partial differential from a range image; a quadratic .partial-differential 
calculating program 204 for calculating a quadratic partial differential from a range image; a local linear- 
plane approximating program 205 for approximating a range image as a local linear plane and calculating 
the coefficients of the equation thereof; a local quadratic-surface approximating program 206 for approximat- 

;5 ing a range image as a local quadratic surface and calculating the coefficients of the equation thereof; a 
Gaussian curvature calculating program 207 for calculating Gaussian curvature; a mean curvature calculat- 
ing program 208 for calculating mean curvature; a normal curvature calculating program 209 for calculating 
normal curvature; a radius-of-curvature calculating program 210 for calculating radius of curvature; a 
Gaussian-curvature sign discriminating program 211 for discriminating the sign of Gaussian curvature; a 

20 mean-curvature sign discriminating program 212 for discriminating the sign of mean curvature; a surface 
category discriminating program 213 for discriminating the category of a surface based upon a combination 
of the sign of Gaussian curvature and the sign of mean curvature; a normal vector calculating program 214 
for calculating a unit normal vector from a linear partial-differential image; and a roof-edge calculating 
program 215 for calculating a roof edge from a unit normal vector. The processing associated with each 

25 program will be described later. 

The data stored in the data memory 2 shown in (b) of Fig. 3 are as follows: an input range image 221 
inputted by a range-image input unit or the like; a jump-edge image 222 detected from a range image; a 
linear partial-differential image 223 detected from a range image; a quadratic partial-differential image 224 
(jetected from a range image; a coefficient image 225 detected from a range image; a calculated Gaussian- 

30 curvature image 226; a calculated means-curvature image 227; a calculated Normal-curvature image 228; a 
calculated radius-of-curvature image 229; a normal-vector image 230 representing a calculated unit normal 
vector; a calculated Gaussian-curvature sign image 231; a calculated mean-curvature sign image 232; a 
calculated surface-category image 233; and a calculated roof-edge image 234. 
Fig. 4 is a flowchart illustrating the flow of this embodiment. 

35 The flow of overall processing of this embodiment will now be described based upon the flowchart 
shown in Fig. 4. 

With regard to the start of processing, it will be assumed that a range image has already been inputted 
by measuring an object using the laser range finder 7, for example, and that the image has been stored in 
the data memory 2. It should be noted that the range image to undergo processing is not limited to one 

-JO based upon such actual measurement but may be one composed of artificial data as obtained from an 
image data base or the like. For a detailed discussion of a method of acquiring a range image, see 
"Measurement of Three-Dimensional Images" by Iguchi and Sato: Shoshodo , (1990). 

First, as initial processing, the stored range image is extracted (step S301) and the image is displayed 
on the display unit (step S302). Next, the type of range image is converted and normalization is performed 

45 along each coordinate axis (step S303). The range image thus converted is used in calculations set forth 
below. This is followed by detection of jump edges and the display thereof (step S304, S305). Linear and 
quadratic partial differentials are calculated and the values thereof are displayed (steps S306 ~ S309). 
Application is made to a local linear plane and a quadratic surface, and the results thereof are displayed 
(steps S310 - S312). Gaussian curvature, mean curvature, normal curvature, radius of curvature and unit 

50 normal vector are calculated using the calculated values of the partial differentials (steps S313, S3 14; S317 
- S320; S327 - S330). Further, sign maps are created from the curvatures obtained, and the maps are 
displayed (steps S321 - S324). Based upon the values of the signs, the surface category is discriminated 
and displayed (steps S325, S326). Roof edges are detected from the calculated unit normal vector, and 
these are displayed (steps S315, S316). 

55 Each of these processing operations will now be described in detail. 

Normalization along each coordinate axis (step S303) 
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A range image is outputted, which image has been subjected to a density conversion in such a manner 
that actual length corresponding to one quantum unit along the x axis is made to agree with the actual 
length of one pixel along the x axis of the inputted image and the actual length of one pixel along the y axis 
of the inputted image. Each pixel of the output image possesses a value of float. 

Let the actual length corresponding to one pixel along the x direction and the actual length correspond- 
ing to one pixel along the y direction both be Ax, and let the actual length corresponding to one quantum 
unit along the z axis be Az. The value Z\ of each pixel of the inputted range image is converted to the 
following zo (x,y): 

zo (x,y) = Zi (x.y) x ^ (0) 

In subsequent processing, the converted z is used. 

Jump-edge extraction (step S304) ' 

A portion in which distance from a point of view is discontinuous is detected in the input image. A 
portion for which the maximum value of the distance difference is in the neighborhood of 8 and is greater 
than a threshold value is detected. 

Letting threshold represent the threshold value. JumpEdge in the following equation is given as the 
value of each pixel: 

JumpEdge(x,y) = Max {|Z(x,y)-Z(x + k,y + 1)|: -l^k, 1^1} (1) 

Fitting based upon local plane (step S310) 

A surface is approximated as a local linear plane and the parameters of the equation thereof are 
calculated. More specifically, each coefficient is calculated by a least-square approximation, in which the 
object surface S of the range image is S = [x,y,-Z(x,y)] and a window of winsize^ is used for the plane 
ax + by +c = -z. 

The operator for calculating each coefficient is decided in the following manner: Let winsize be the 
window size (the number of pixels constituting one side of a square window) of the operator; let this be an 
arbitrary, positive odd number. 

A • C = B (2) 

where it is assumed that the following holds: 



A = 



xl 
x2 



yl 1 
y2 1 



B 



-zl 



c = b 



a 



d 



xn 



yn 



1 



-znl 



in which case we have 
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C = (At-A)~l • At-B 
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In view of a window of size winsize = (2w + 1) sliown in" Fig. 5, we fiave thie following if the origin 
talon at the center of this window. 



C = 



l/3w(w+l) (2w+l)^ 0 

0 l73w(w+l) (2w+l) 

0 0 



0 
0 

(2w+l) 



A^-B 



(4) 



= l/w(w+l) {2w+l) 2 



3 
0 
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0 
3 
0 



0 
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w (w+l) 



A^-B 



(5) 



Letting 1 /w (w+l ) (2w+l ) 2 



3 
0 
0 



0 
3 
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0 
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w ( w + 1 ) 



= E 



hold and letting each element be represented by (eij) 

... (6) 



we have 
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C = E-At-B = (eij) 
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1 



X2 
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e22*yi 

€33 



eii*X2 
e22*y2 

633 



eii*x„ 

622 *yn 
©33 



B 



75 



20 



25 



More specifically, we- have 



II II n 

= eiiX Xi- (-Zi) b = 622^ yi* ("^i) c = eaaX " 



i-l 



i-1 



i-1 



(8 



A plane estimation operator of winsize = 3, 5 (3'3, 5'5) Is illustrated as an example, 
(a) In case of window size 3*3 (winsize = 3, w = 1) 



w = 1 is substituted into Eq. (6) to obtain E (namely en, 622, 633). and (xi, xa, Xn), (yi . yi. 
yn) are substituted into Eq. (8) to obtain the following operator. This operator is used to multiply B (namely 
30 the range image). 



35 
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(b) In case of window size 5'5 (winsize = 5, w = 2) 

The following operator is obtained in the same manner as in (a) above: 



55 
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1/25' 



11111 
11111 
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11111 
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20 

Fitting based upon local quadratic surface (step S312) 

The object surface S = [x,y,-Z(x,y)] of the range image is fitted to a quadratic surface 
ax^ + by2 +cxy + dx + ey + f = -z by least-square approximation, and each coefficient is calculated. 
25 The operator for calculating each coefficient is obtained through an approach the same as that used in 
fitting based upon a local plane described above. Though a detailed description is omitted, two types of 
operators are shown in Fig. 6. 

Linear' quadratic partial differentials (steps S306, S308) 

A linear or quadratic partial differential is calculated from a range image by the following three methods: 
(1) the partial differential is obtained by directly applying a partial-differential operator; (2) the partial 
differential is obtained from coefficients of a local plane equation which has undergone fitting; (3) the partial 
differential is obtained from coefficients of a local quadratic-surface equation which has undergone fitting. 

(1) Function for obtaining partial differential directly from original image 

Two types of partial-differential operators are illustrated in Fig. 7 by way of example. 

40 (2) Function for obtaining linear partial differential from coefficients of local plane which has underdone 
fitting 

The differential value of a certain point of interest is calculated through the following method: 
Let n, m represent the operator size (the number of pixel constituting one side of a window). As shown 
in Fig. 8, n X m windows having a size of n x m and containing the pixel of interest are prepared, and these 
windows are made to approximate -Z(x,y) = ax + by + c in accordance with the method set forth in the 
description of fitting based upon a local quadratic plane. A window for which a variance in the absolute 
value of an error is minimum is selected from the n x m windows, and the differential value of the pixel of 
interest is obtained from the equation of this window. In case of a plane, the following equations are 
obtained when the partial differential value is calculated from the equation ax + by + c = -z of a fitted 
plane: 

Zx = -a 
Xy = -b (9) 

55 

(3) Function for obtaining partial differential from coefficients of local quadratic surface which has underdone 
fitting 
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The differential value of a certain point of interest is calculated through the following method: 
Let n, m represent the operator size (the nunnber of pixel constituting one side of a window). As shown 
in Fig. 8, n x m windows having a size of n x m and containing the pixel of interest are prepared, and these 
windows are made to approximate -Z(x,y) = ax + by + c in accordance with the method set forth in the 
5 description of fitting based upon a local quadratic surface. A window for which a variance in the absolute 
value of an error is minimum is selected from the n x m windows, and the differential value of the pixel of 
interest is obtained from the equation of this 'window. In case of a plane, the following equations are 
obtained when the partial differential value is calculated from the equation ax^ + by^ + cxy + dx + ey + f 
= -z of a fitted plane: 

w 

Zx = -2ax 
Zy = -2by 
Zxx = 2a 
Zyy = 2b 
75 Zxy = c 

Calculation of normal vector (step S313) 

The normal vector of a local tangent plane of a surface is calculated as a unit vector of size 1 . 
20 The input image is a linear partial-differential image. 

The X, y and z components of the unit normal vector are calculated based upon the following equations: 

Unit normal vector (x,y,z) = (Ni,N2,N3) (11) 

25 Ni = Zx/VZx^ + Zy^ + 1 
N2 = Zy/v'Zx'^ + Zy^+l 
N3 = -1/VZx^ + Zy^ + 1 

where Zx represents a linear partial differential based upon the x component of the distance data, and Zy 
30 represents a linear partial differential based upon the y component of the distance data. 
Eq. (11) is based upon computation the following computation; 
The surface of a range image handled by this technique is 

S = [x,y,-Z(x.y)] 

35 

A normal vector indicated by the following equation is obtained as the outer product of a linear partial 
differential Sx = (1,0.-Zx) based upon the x component of S and a linear partial differential Sy = (1,0.-Zy) 
based upon the y component of S: 

40 N = (Zx,Zy,-1) (12) 

If this is adopted as a unit vector of magnitude 1, Eq. (11) is obtained. 

Roof-edge detection (step S3 15) 

45 

An inflection point at a distance from a point of view is extracted. The input image is a normal-vector 
image. 

A portion for which the maximum value of the angular difference of the unit-normal vector is in the 
neighborhood of 8 and is greater than a threshold value is detected. RoofEdge in the following equation is 
50 given as the value of each pixel: 

RoofEdge(x.y) = Max {|N(x.y)*N(x + k,y + 1)|: -l^k. 1^1} (13) 

where N(x,y) represents the following normal vector: 

55 

N(x.y) = [Nt(x.y). N2(x,y). N3(x,y)] 

Ni(x,y) is the < -component image of the vector in the unit-normal direction; 



cy - d 
■ cx - e 



(10) 
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N2(x.y) is the y-component image of the vector in the unit-normal direction; 
N3{x.y) is the z-component image of the vector in the unit-normal direction 



Gaussian curvature calculation (step S317) 



Each partial differential value is inserted into the following equation to calculate the Gaussian curvature 

K: 



where Zx represents the linear partial differential based upon the x component of the distance data; 
Zy represents the linear partial differential based upon the y component of the distance data; 
Zxx represents the quadratic partial differential based upon the x component of the distance data; 
Zyy represents the quadratic partial differential based upon the y component of the distance data; and 
Zxy represents the quadratic partial differential based upon the x, y components of the distance data. 

Mean curvature calculation (step S319) 

Each partial differential value is Inserted into the following equation to calculate the mean curvature H: 



where Zx represents the linear partial differential based upon the x component of the distance data: 
Zy represents the linear partial differential based upon the y component of the distance data; 
Zxx represents the quadratic partial differential based upon the x component of the distance data; 
Zyy represents the quadratic partial differential based upon the y component of the distance data; and 
Zxy represents the quadratic partial differential based upon the x, y components of the distance data. 

Creation of sign map of Gaussian curvature (Steps S321, S322) 

Let kimg(x,y) represent an inputted Gaussian-curvature image, and let thk represent a threshold value 
(float) which decides the sign. 

An image in which the following classes are given for each of the pixels is outputted: 
kimg(x.y) < -thk is Class 1 where the Gaussian curvature is regarded as being negative; 
-thk<kimg(x,y) < thk is Class 2 where the Gaussian curvature is regarded as being zero; and 
kimg(x,y) > thk is Class 3 where the Gaussian curvature is regarded as being positive- 
Creation of sign map of mean curvature (Steps S323, 8324) 

Let himg(x,y) represent an inputted mean-curvature image, and let thh represent a threshold value 
(float) which decides the sign. 

An image in which the following classes are given for each of the pixels Is outputted: 
himg(x.y) < -thh Is Class 1 where the mean curvature Is regarded as being negative; 
-thh<himg(x,y) < thh is Class 2 where the mean curvature is regarded as being zero; and 
hlmg(x.y) > thh is Class 3 where the mean curvature is regarded as being positive. 

Surface classification (steps S325, S326) 

Surfaces are classified into eight surface types based upon the theory of differential geometry. Here 
thk. thh represent threshold values (float) which decide sign. A detailed discussion of the differential 



Gaussian curvature K = 



Zxx^Zyy-Zxy 
(l+Zx+Zy) 2 



(14) 



Mean curvature H 



Zxx+Zyy-<-Zxx*Zy+Zyy*Zx-2*Zx*Zy*Zxy 
(1+Zx+Zy) 2/3 



(15) 
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geometry of curved surfaces is given if the following literature: 

"An Outline of Differential Geometry" by Chuji Adachi, Baifukan. (1976); 

"Differential Geometry of Curved Lines and Curved Surfaces" by Shoshichi Kobayashii, Shokado, 
(1977); and 

5 "Introduction to Differential Geometry and Topology" by Honma and Okabe, Basic Mathematics Series 

6. Shinyosha, (1979). ; 

Examples in which this is utilized in segmentation' of a range image are as follows: 
"A Hybrid Technique for Segmenting a Range Image based upon Differential Geometric Characteris- 
tics" by Naokazu Yokoya and Martin D. Levine, Joshoron, Vol. 30, No. 8, pp. 944 - 953 (1989); 
10 "Surfaces in Range Image Understanding" by Paul J. Besel, Springer-Verlag, (1988); and 

"Invariant Surface Characteristics for 3D Object Recognition in Range Images" by Paul J. Besel and 
R.C. Jain. CVGIP, 33. pp. 33 - 80. (1986). 

An image in which the following eight classes are given for each of the pixels based upon the sign of a 
Gaussian-curvature image kimg(x,y) and a mean-curvature image himg(x,y) is outputted: 
15 kimg(x,y) > thk and himg(x,y) < -thh is Class 1; 
kimg(x,y) > thk and himg(x,y) > thh is Class 2; 
-thk < kimg(x,y) < thk and himg(x.y) < -thh is Class 3; 
-thk < kimg(x,y) < thk and himg(x,y) > thh is Class 4; 
-thk < kimg(x,y) < thk and -thh < himg(x,y) < thh is Class 5; 
20 kimg(x,y) < -thk and -thh < himg(x,y) < thh is Class 6; 
kimg(x,y) < -thk and himg(x,y) < -thh is Class 7; 
kimg(x,y) < -thk and himg(x.y) > thh is Class 8; and 
kimg(x,y) > himg(x,y)2 (forbidden area) is Class 0. 

A point in Class 1 is referred to as a negative elliptical point (K>0, H<0), a point is Class 2 as a positive 
25 elliptical point (K>0. H>0), a point is Class 3 as a negative parabolic point (K = 0, H<0), a point is Class 4 as 
a positive parabolic point (K = 0, H>0), a point in Class 5 as a fiat point (K = 0. H = 0), a point in Class 6 as a 
minimum point (K<0, H=0), a point in Class 7 as a negative hyperbolic point (K<0. H<0), and a point in 
Class 8 as a positive hyperbolic point (K<0, H>0). These curved surfaces are shown in Fig. 9. 

30 Calculation of normal curvature (step S327) 

Normal curvature along each of the x and y axes when cut by a plane containing straight lines that are 
parallel to the z axis and either the x axis or y axis is calculated for each pixel. 

Each pixel value of inputted linear and quadratic partial-differential images is substituted into the 
35 following equations to calculate normal curvatures x^,xy along the respective x and y directions: 

Normal curvature Kx 



Zxx 



(1+Zx) 2/3 



(1+Zy) 2/3 



(16) 



where Zx represents the linear partial differentia! based upon the x component of the distance data; 
Zy represents the linear partial differential based upon the y component of the distance data; 
45 Zxx represents the quadratic partial differential based upon the x component of the distance data; 
Zyy represents the quadratic partial differential based upon the y component of the distance data. 

Calculation of radius of curvature (step S329) 

50 Radius of curvature (1 'normal curvature) along each of the x and y axes when cut by a plane containing 
straight lines that are parallel to the z axis and either the x axis or y axis is calculated for each pixel. 

Each pixel value of inputted linear and quadratic partial-differential images is substituted into the 
following equations to calculate radii of curvature px.py: 

55 
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Radius of curvature Px 



1/K 



( 14-Zx) 2/3 
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Zxx 
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where Zx represents the linear partial differential based upon the x component of the distance data; 

Zy represents the linear partial differential based upon the y connponent of the distance data; 

Zxx represents the quadratic partial differential based upon the x connponent of the distance data; 

Zyy represents the quadratic partial differential based upon the y connponent of the distance data. 

In accordance with the first enribodiment. as described above, by providing a step of converting the type 
of a range image and normalizing each axis, pixel values are calculated in conformity with the type of data, 
which is given for each pixel of an inputted range image, using parameters such as aspect ratio along each 
axis of the coordinate system. As a result, the relationship between the coordinate system in which an 
object is placed and the coordinate system of the image need not be stipulated in advance, and it is 
unnecessary to give parameters such as the aspect ratio along each of the coordinate axes each time 
processing is performed. 

(Example in which entirety of three-dimensional model is obtained from partial range image) 

Fig. 10 is a diagram showing the construction of a memory according to a second embodiment of the 
invention. 

Programs stored in the program memory 1 are a range-image display program 111 for three- 
dimensionally displaying an inputted range image, a main-axis designation and user interface program 112 
for designating a main axis of rotation with regard to a displayed range image, a shape-model constructing 
program 113 for constructing a three-dimensional model inclusive of a background (rear) surface in 
response to input of the range image and main axis, and a shape-model display program 114 for 
performing a rendering operation by applying lighting and shading, etc., to the obtained three-dimensional 
model. 

The data stored in the data memory 2 include range-image data 121 inputted from a range-measuring 
sensor or range-image DB, and three-dimensional data 122 constructed by the processing of this 
embodiment. 

A simplified flowchart of the algorithm according to this embodiment will be described with reference to 
Fig. 11. 

First, at step SlOl, data having the aforementioned range-image format are obtained from a three- 
dimensional measuring unit (not shown) or range-image DB (not shown) for measuring an object based 
upon a range-image measuring technique such as light time-of-fltght measurement or triangulation. With 
regard to measurement of the range image, there are central projection and cylindrical projection types of 
systems besides the above-mentioned parallel-projection system. In the present step, a conversion is made 
to the coordinate space of one projection system (the parallel-projection system) and subsequent process- 
ing is unified. A point at which range is greater than a fixed value is taken as infinity in order to be regarded 
as background; it is not regarded as part of an object. The characterizing feature of this embodiment 
resides in the processing after acquisition of data, and therefore the above-described processing is not 
described in detail. 

Since the range image obtained basically has a lattice structure, at step Si 02 the diagonals of the 
lattice shape are connected to convert the range data into a row of triangular polygons, as shown in Fig, 7, 
and a parallel projection is made upon a two-dimensional plane using graphics library such as PHIGS. 

At step Si 03, the main axis of an object having axial symmetry is decided after the three-dimensional 
object is thus displayed in two-dimensional space. However, since depth information cannot be accurately 
reflected in two-dimensional space, the main axis is decided using an image parallel-projected upon three 
surfaces (top surface, front surface and side surface). This technique will be described with reference to Fig. 
12. 

Fig. 12 shows a drawing of three views obtained when a cylindrical object is parallel-projected from the 
+ O0 direction along the z direction. The three-dimensional coordinate system employs a right-handed 
coordinate system. As one example of deciding a main axis, first the top view of the three-surface view is 
utilized and points 41, 42 are clicked by a mouse to specify the position of x.z. The two points thus 



17 



EP 0 526 881 A2 



obtained similarly are reflected as points 41', 42' in the front and side views. 

As the next step, point 41* is selected first in the front view and is dragged (by dragging the mouse 5) 
to the actual position 41". whereby the y-coordinate value is designated. When the point 42' is similarly 
dragged to point 42", the straight line connecting the points 41" and 42" serves as the main axis of 

5 rotation. This designation and dragging of points can readily be achieved by designating a mode using a 
pop-up menu, for example, in a window system. 

The next step Si 05 is a central step for constructing a three-dimensional model from the main axis and 
the range data. This algorithm will be described using Fig. 13. The three-dimensional shape constructed by 
the algorithm of this embodiment is assumed to be a polygon model. An example of a simple cube is 

w illustrated in Fig. 14 as an example of the data structure. In Fig. 14. (a) shows an example of the data 
structure, and (b) expresses the data structure of (a) as a cubic shape. Here xo, yo, zo represent points of 
intersection between the surface of the cube and each axis. 

The data structure of this embodiment is broadly divided into two portions. The first half of Fig. 14(a) is 
a portion in which vertex information constituting a polygon designated. The essential data are the (X,Y,Z) 

15 values of the coordinates of the vertices, such as (-1.0.-1.0,-1.0) in Fig. 14(a), and the values (Nx,Ny,Nz) of 
the normal vector at an optional vertex, such as (-0.57735,-0.57735,-0,57735) in Fig. 14(a) . The second-half 
portion indicates the connectivity of ridge lines constituting the polygon with respect to the designated 
vertex Information, which is phase information. There are initially n-number of ridge lines (four in the 
quadrangle of Fig. 14(a) ) constituting the polygon, and these are followed by index values to n-number of 

20 index vertices constituting a polygon. [Indices of 0 - n-1 are attached in order to the vertex coordinates of 
the first half, and the indices of the vertices constituting the polygon are placed in a row, such as (0, 1, 2, 3) 
in Fig. 14(a). These are indicated as points 0 through 7 in the cubic shape of Fig. 14(b).] In the final data 
structure of the cube, a plurality of polygons in a plane area combined and the polygons form a quadrangle. 
In Fig. 14(b), a cube Is illustrated for the sake of simplicity. However, It is obvious that a three-dimensional 

25 model such as a cylinder is formed in the same manner; the amount of data Is merely greater and the data 
structure is just more complicated. 

At step S501, phase information, which is connectivity information regarding the vertices, is constructed 
after the extraction of vertex information of a part of the object indicated by the black dots In the range 
image on the sensor side of the obtained lattice shape (see Fig. 15, which is an example in which resolution 

30 Is reduced In order to simplify the description). Basically, the polygon created by the algorithm of this 
embodiment is made a triangle owing to the ease of processing. When the phase Information is 
constructed, processing is executed every scanning line, as shown in Fig. 15. This will be described In 
somewhat greater detail. 

The processing Indicated below is executed by scanning pixel values in order from the smaller values 

35 of X in the direction of scanning lines 1, 2, n along the y direction. Fig. 17 illustrates, in the form of 
pseudo-coding, the algorithm for when ridge-line phase information is constructed by scanning the value of 
a p-th pixel In the x direction on an n-th scanning line. In Fig. 17, let depth (p.n) represent the range pixel 
value (depth) at a pixel (p.n) in two dimensions. The procedure of the algorithm shown in Fig. 17 is the 
flowchart of Fig. 16 when expressed in simplified form. 

40 It is determined at step S81 whether the present pixel of Interest is on an object. If the pixel Is on the 
object, the program proceeds to steps S82, S83, where It is determined whether a pixel to the left or right of 
the pixel of interest lies In the background. If It does, then the pixel of interest Is registered as the contour of 
the object at step S84. Next, at step S85. a "1" flag is set if four pixels inclusive of the pixel of interest and 
the pixels below, to the right and to the lower right of It lie on the object, and a "0" flag Is set if these pixels 

45 tie in the background. Then, at step S86, a polygon (a triangle In this embodiment) is created and registered 
in correspondence with the number of pixels for which the flag Is "1". First, in a case where the number of 
pixels for which the flag Is "1 " two or less. It is judged that there is no polygon and nothing is registered at 
step S87. In a case where the number of pixels for which the flag is "1 " three, it judged that there is one 
polygon and these three "1" pixels are registered as the indices of vertices at step S88. In a case where 

50 the number of pixels for which the flag Is "1 " four, It judged that there are two polygons. Then, at step S89, 
the pixel of interest and the pixels below it and to the right are judged as defining one polygon, the pixels to 
the lower right, below and to the right are judged as defining one polygon, and the Indices of the vertices 
are registered. In this example, minute polygons of pixel units are considered, but the polygons may be 
coarse polygons in conformity with the precision. 

55 The phase information obtained in information indicative of the front surface as seen from the sensor. At 
step S503, position information Indicative of the back surface is computed from the measured range-pixel 
values and the information indicative of the main axis. 

Specifically, let (xi.y^.zi) and (x2.y2.22) represent two points which decide the axis of rotation, and let 
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(Xp.Yp.Zp) represent a range value already known. Then, t in Eq. (18) below is solved, and the solution is 
utilized to obtain a point (Xc.yc.Zp) = [xj +t(x2-xi ). yi+t(y2-yi). z; +t(22-2i )), which is the point of 
intersection between the straight line (xKy,,zi). (X2.y2,22) and the perpendicular to this straight line from the 
point (Xp.yp.Zp). A position (2xc-Xp, 2yc-yp, 22c-Zp) on the back surface, which is a synnmetrical point of 
5 (Xp,yp,Zp) with regard to (Xc.yc.Zp), is calculated, and this is appended as vertex infornnation of the polygon. 



At step S504, a series of indices of vertices forming the peripheral contour from the index values of the 
front-surface vertices corresponding to the back-surface vertices are stored internally at the same time, and 
phase information of the back surface is created utilizing the phase information obtained at step S503. 
Periphery information, which is a series of indices of vertices forming the peripheral contour of the back 

20 surface, also is stored in the same manner as at step S502, 

Next, in order to connect the outer periphery of the front surface and the outer periphery of the back 
surface at step S505, the area of the triangle obtained by the connected lines is treated as an evaluation 
function using dynamic programming, and the front and back surfaces are connected so as to minimize the 
cost. As a result, the triangle obtained by the connection is appended to the information indicative of the 

25 polygon model. Thus, the shape model of a solid three-dimensional object having axial symmetry is 
created. 

Finally, at step Si 06, the three-dimensional model [see Fig. 14(a)] obtained at step S105 is interpreted, 
after which a graphics command is issued to project the three-dimensional shape onto a two-dimensional 
plane and verify the cubic shape. In the algorithm of this embodiment, this function is implemented by 
30 using a graphics library such as PHIGS. Examples of processing specific to three-dimensional processing at 
such time are lighting of the light source, shading, elimination of concealed lines and concealed surfaced, 
the depth cue effect, etc. 

In accordance with this embodiment, shape data of an unknown portion can be furnished based upon a 
range image obtained from one direction. Consequently, a three-dimensional shape can be inputted and re- 
35 utilized in simple fashion without expensive modeling software such as CAD software. 

(Example of display of three-dimensional model) 

Fig. 18 is a diagram showing the construction of a memory according to a third embodiment of the 
40 present invention. 

The programs stored in the program memory 1 include a calculation program 31 1 for calculating the 
sense of a surface from shape data, and a display program 312 for displaying the sense of the surface 
obtained. The data stored in the data memory 2 include shape data 321 inputted from a three-dimensional 
shape input unit, a three-dimensional model creating unit or a three-dimensional shape data base, and 
45 sense data 322 indicating the sense of the surface of an object obtained by the processing of this 
embodiment. 

The items of data mentioned above will be described before discussing the algorithm of the processing 
according to this embodiment. The shape data 21 represent the position of the object surface in three- 
dimensional space. Specifically, the data represent the coordinate positions of points on a surface, a 
50 mathematical expression representing the object or coordinate positions of characteristic points if the object 
is a geometrical object, the vertices of a polyhedron, phase information, etc. The data 322 indicative of the 
sense of a surface represent the sense of the object surface in numerical form. In this embodiment, a unit 
normal vector is used. There is sense with regard to a reference point or sense with regard to line of sight 
in three dimensions. 

55 The general flow of the algorithm of this embodiment will now be described with reference to Fig. 19. 

First, at step Sill, the shape data 321 of an object is obtained from a three-dimensional shape input 
unit, a three-dimensional shape-data creating unit or a three-dimensional shape data base. Furthermore, this 
embodiment is characterized by the processing which follows acquisition of the shape data 321, and the 



10 



X2-X1 

y2-yi [xi+t (X2-X1) -xpyi-i-t (y2-yi) -ypzi+t (22-21) -2p)=o 
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creation of this shape data 321 will now be described here. Next, at step S112, the unit nornnal vector at 
each point on the surface of an object is calculated from the shape data 321. Algorithms for calculating a 
unit normal vector are described In detail in "A Hybrid Technique for Segmenting a Range Image based 
upon Differential Geometric Characteristics" by Naokazu Yokoya and Martin D. Levine, Joho Short Gakkai 

5 Ronbunshi, Vol. 30, No. 8, Aug. 1989. One of these is used here. Next, at step S1 13, the unit normal vector 
is converted into data capable ;of being identified, and this is displayed on a two-dimensional plane. 

One type of unit normal vector present in an image possesses information indicative of sense, number 
and distribution. These items of information are expressed by a combination of symbols such as characters, 
numerals, arrows and dots and attributes such as the accompanying color, size, length and display position. 

w This is a conversion (referred to also as symbolization) to identifiable data referred to in this invention. The 
displayed information, symbols and attributes can be selected by the user in dialog fashion depending upon 
the purpose of use. 

Examples of symbolization of a unit normal vector at step Si 13 are illustrated in (a), (b), (c) of Fig. 20 
and (a), (b) in Fig. 21. Fig. 20' shows examples in which the sense of an object surface is displayed as the 
15 length and direction of each line segment, Fig. 21 A shows examples in which the sense of an object surface 
is displayed as the color of each point, and Fig. 21 B shows original color images corresponding to Fig. 21 A. 

The algorithm for obtaining Fig. 20 will be described. 

Let (nx.ny.nz) represent the unit normal vector of a point (x,y) on a range image. Further, let s represent 
the sampling interval of the range image. In a case where a range image is seen from a direction parallel to 

20 the Z axis at the middle of the image, the unit normal vector can be expressed as a straight line connecting 
point (x,y) to point (x + n^^y + ny). However, the straight line is very short and its direction is difficult to 
distinguish under these conditions. Accordingly, the normal vector is multiplied s times in advance. At this 
time, the unit normal vector is a straight line from point (x,y) to point (x + snx,y + sny). 

Furthermore, when it is attempted to observe a range image from a different direction, the image should 

25 be rotated about the X and Y axes by angles of a and jS, respectively. Let z represent the height, from the 
X-Y plane, of a point (x,y) on an image at this time. In such case. (x,y) is converted into [xcos/3 + ysinjS, - 
zsina - {xsinj3iycos;3)*cosa], and (x + sn^.y + sny) is converted into [(x + snx)'cos^ + (y + sny)*sinj3, - 
(z + snx)*sina - {(x + snx)*sin^ - (y + sny)*cos;3}'cosa]. Accordingly, the normal vector in this case is a 
straight line connecting two points obtained after the points (x,y) and (x + sn^.y + sny) have been connected. 

30 An algorithm for converting a unit normal vector into color for in instance in which Figs. 21 A and 218 
are displayed will be described as an example of symbolization. Fig. 22 is the flowchart in simple form. 

At step 8201, the normal vector given by the rectangular coordinates (x.y.z) is transformed into a 
normal vector in polar coordinates {^,^,<P). where d represents the angle of rotation (0 to 360 ' ) of the normal 
vector about the z axis, and 6 represents the angle (0 to 90') of the normal vector from the x-y plane. 

35 Next, the polar coordinates {^,Q,<t>) are transformed into HLS color-space coordinates (h,0.5,s) at step 

S202. The transformation equation is h = s = ^/'90* , where h represents hue and varies from 0 to 360' , 
blue is 0'. magenta is 60', red is 120', yellow is 180*, green is 240* and cyan is 300'. If we let a 
represent the hue of a color, then (a + ISO' )mod360' will represent the complimentary hue thereof. Further, 
s represents saturation and varies from 0 (achromatic) to 1.0 (the most brilliant color for a given luminance). 

40 The colors having the highest saturation in a plane of equal hues are arrayed on a hue circle for which L 
(luminance) is 0.5. 

Finally, the HSL color-space coordinates (h,0.5,s) are transformed into RGB color-space coordinates 
(r.g.b) at step S203, and the point at which the unit normal vector is present is displayed in this color. With 
regard to an algorithm for transforming the HSL color-space coordinates (h.0.5,s) into RGB color-space 

45 coordinates (r,g,b), a detailed description is given, for example, in "Computer Imaging", Television 
Gakkaihen, Corona Publishing Company. One of these is used. 

The examples of Fig. 21 A and original color images of Fig. 21 B thus displayed will now be described in 
further detail. In Fig. 21 A, numerals 43a, 43b denote background, for which an achromatic color exhibiting 
tow luminance is selected, by way of example. 

50 In Fig. 21 A, numerals 41a - 41 g denote planes. Therefore, hue varies in correspondence with sense of 
the plane about the z axis, and saturation varies in correspondence with the angle of the plane from the x-y 
plane, though this is constant in one plane. For example, the planes 4la, 41b, 41c, 41d have different hues 
but the same saturation. Depending upon the difference in hue. it can be determined whether the plane 41e 
is concave or convex. Since saturation represents the slope of the plane, the degree of convexity or 

55 concavity of the plane 41 e can be determined. These determinations are not made by storing color. Rather, 
assignment of hue, saturation and luminance that will enable a determination to be made from the 
impression of a displayed image is desirable. 

Numerals 42a - 42c denote curved surfaces. In surfaces having the same sense about the z axis, hue 
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is the same but saturation varies. For example, in case of a spherical surface 42a, points 42ai and 42a2 
have the same hue but the saturation diminishes as point 42ai is approached from point 42a2. Points 42ai 
and 42a3 have different hues but the same saturation. The original color images of Fig. 21 B demonstrate 
the above description. Here also, judgment of shape is not made by storing color; assignment of hue, 

5 saturation and luminance that will enable a determination to be made from the impression of a displayed 
image is desirable. Accordingly, the assignment of hue and saturation in the HLS color-space coordinates 
(h,0.5,s) in the above-described embodiment is one example thereof. It will suffice to select the optimum 
assignment corresponding to the objective and application. 

This embodiment is described taking a display as an example. However, it is obvious that the same 

10 effects can be attained even if the invention is applied to output to a printer or the like. Further, the 
embodiment has been described using a unit normal vector as an indication of the sense of a surface. 
However, if the data are such that sense is capable of being identified, these data can be used. Though a 
conversion is made from a unit normal vector to display data, it is also possible to make a direct conversion 
to display data. 

15 In accordance with this embodiment, the sense of any point on the surface of an object in three- 
dimensional space is outputted in an identifiable manner. As a result, all surface shapes of objects can be 
outputted uniquely based upon a single two-dimensional image viewed from any direction, and the 
operations, number of steps and complicated processing necessary for recognizing the shape of an object 
can be reduced. In addition, the precision with which the shape of an object is recognized can be raised by 

20 selecting information in line with the objective and the data conversion best suited for the information. 

(Example of creation of three-dimensional mode! based upon polygonal patches) 

Fig. 23 is a diagram illustrating the construction of a memory according to the fourth embodiment, and 
25 Fig. 24 is a diagram showing a flowchart of overall processing according to the fourth embodiment. 

In Fig. 23. (a) illustrates the programs stored in the program memory 1 of this embodiment. The 
programs include a range-image input processing program 401 for inputting a range image to the 
apparatus, a surface classifying program 402 for creating a surface category map from the given range 
image,- and a polygonal patch model creating program 403 for creating a polygonal patch model from a 
30 range image based upon the surface category map. 

Fig. 23(b) shows the construction of the data memory 2 in this embodiment, in which numeral 41 1 
denotes an inputted range image. 412 a created category map and 413 a created polygonal patch model. 

The flow of overall processing of this embodiment will now be described based upon the flowchart 
shown in Fig. 24. First, at step S21 , a range image is inputted as by measuring an object by the laser range 
35 finder 7 in accordance with the range-image input processing program 401. and the range image is stored 
in the data memory 2 as the range-image data 411. This input is not limited to one obtained by actual 
measurement, for an arrangement can be adopted in which data stored in an image data base are inputted. 

Next, at step S22, the inputted range-image data 411 (three-dimensional coordinate data of each point 
constituting the surface of the object) are processed in accordance with the surface classifying program 
40 402. The category of the surface constituting the object surface is classified to create the surface category 
map 412. 

In accordance with the polygonal patch model creating program 403, the polygonal patch model 413 is 
created at step S23 by polygonal patches best suited for the classified surface category based upon the 
range-image data 411 and surface category map 412. 
45 Fig. 25 is a flowchart showing the details of step S22, namely the processing for surface classification. 

First, at step S31, a range image having a range value for each pixel of the image is inputted as a two- 
dimensional array of three-dimensional coordinate data, whereupon estimation of a local curved surface is 
performed at step S32 by least-square approximation using a window of a suitable size. This is followed by 
step S33. at which first and second differentials at each pixel are calculated from the estimated local curved 
50 surface or by another method. Then, at step S34, Gaussian curvature and mean curvature are calculated, 
followed by step S35, at which a curvature sign map is created from the signs of the two curvatures 
obtained. Finally, at step S36, classification is made into eight surface categories shown in Fig. 26 
depending upon the sign, and a surface-class image is generated in which the respective codes (the 
surface-class numbers) serves as pixel values. 
55 An algorithm for curved-surface classification described in brief here is disclosed in the paper "A Hybrid 
Approach to Range Image Segmentation Based on Differential Geometry" by Naokazu Yokoya and Martin 
D. Levine, Joho Shori GakHai Ronbunshi, Vol. 30, No. 8. Aug. 1989. Here a detailed description is omitted. 

Fig. 27 is a flowchart corresponding to the polygonal patch model creating program 403, and Fig. 28 is 
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a diagram schematically showing an example of processing of range-image data processed by the 
procedure of Fig. 27. 

At step S51 in Fig. 27, the surface-class image [(a) in Fig. 28] created by the procedure described in 
connection with the flowchart of Fig. 25 is inputted, and an image [(b) in Fig. 28] representing one certain 
5 surface class is successively extracted from this surface-class image at step S52. Fig. 28(b) is an image in 
which only location at which the surface class is 5 are extracted, by way of example. 

At step S53, the same label (number) is assigned to all pixels belonging to the same connecting 
component, and a label image [(c) in Fig. 28] is generated in which different labels are assigned to different 
connecting components. 

10 This is followed by step S54, at which a polygonal patch model corresponding to the surface class is 
generated for every label (region number) assigned at step S53. That is, with regard to a region in which 
the surface class is n, a procedure is executed for creating a polygonal patch of each region of a surface for 
which the surface class is classified into n. This procedure will be described in detail later. 

If it is found at step S55 that the next surface class number is that of a class which has not yet been 

IS processed, then processing is repeated from step S52. Processing is terminated when the eighth class 
number ends. 

Thus, steps S52 - S54 are carried out for all extracted surface classes. 

Next, in the surface classes shown in Fig. 26, a procedure for creating a polygonal patch for each 
region of a surface classified as class 5 will be described in accordance with the flowchart of Fig. 29. 
20 Step S71 calls for inputting the results obtained by region-labeling an image which represents a surface 
whose surface class has been classified as class 5 at step S53. 

Step S72 calls for extracting the contour lines of all regions of surfaces whose surface class has been 
classified as class 5, and which have undergone labeling, at step S53 [(d) in Fig. 28]. 

As shown in Fig. 28(d), numbers equal to those assigned at step S53 are assigned to the contours. 
25 Step S73 calls for extracting the contours in numerical order and applying the processing below to each 
one. 

At step 874, the contour is traversed setting out from a certain predetermined starting point, numbers 
are assigned successively to points located every predetermined number of pixels on the contour, and the 
coordinates of these points on the contour are read. The points are referred to as vertices, and the numbers 
30 are referred to as vertex numbers. 

This is followed by step S75, at which the coordinates in three-dimensional space corresponding to the 
coordinates on the image read at step S74 are read from the original range image, and these are written in 
a polygonal patch data table, shown in Fig. 30, as vertex information along with the surface category 
numbers, region numbers (equal to the contour numbers) and vertex numbers. Furthermore, the traversed 
35 vertex numbers are written in the polygonal patch data table in order as phase information. 

If the next contour number is one which has not yet undergone processing ("NO" at step S76), then 
processing is repeated from step S73 onward. When all contour numbers have ended, processing is 
terminated. 

Thus, steps 873 - 875 are carried out with regard to all contours extracted at step S72. 
40 Finally, a polygonal patch data table shown as an example in Fig. 30 is created. 

Polygonal patches are created in the same manner also in regions of a surface for which the surface 
class of Fig. 26 is other than 5(1 or 2 or 3 or 4 or 6 or 7 or 8). 

Fig. 31 is an explanatory view in which the polygonal patch mode! is created. 

A certain point inside an extracted region is extracted in order from smaller values of x in the scanning- 
45 line direction (along the x axis in Fig. 31) and from smaller values of y in a direction (along the y axis in Fig. 
31) perpendicular to the scanning lines, and the processing set forth below is executed. 

The flowchart for extracting the p-th value in the x direction on an n-th scanning line and creating ridge- 
line'phase information is the same as that in Fig. 16 of the second embodiment. An example of pseudo- 
coding is illustrated in Fig. 45. In Fig, 45, a variable x is a value predetermined in accordance with the 
50 surface class. 

In processing of this kind, the value of x is set small for a surface class that is highly irregular, and the 
value of X is set large for a surface class that is flat. As a result, a polygonal patch model can be created in 
which representative points are selected densely at a portion having a complicated shape and sparsely at a 
portion that is smooth. 

55 

(Other example based upon polygonal patches) 

In this embodiment, if, after representative points are selected densely from points of interest, a change 
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in a normal vector between a point of interest and a selected point is less than a predetermined value, the 
representative points are re-selected at greater inten/als. whereby ihe spacing of representative points at 
portions where change is great is narrowed and the spacing of representative points at smooth portions is 
widened. 

5 Fig. 32 illustrates the construction of the program memory 1 and data memory 2 according to the fifth 

embodiment. In Fig. 32, (a) is a diagram showing the programs in the program memory 1 of the fifth 
embodiment. The programs include a range-image input processing program 501 the same as that in the 
fourth embodiment, a normal-vector computation program 502 for obtaining a normal vector by spatially 
differentiating the. range-image data applied thereto, and a polygonal patch model creating program 503 in 

w accordance with this embodiment. 

In Fig. 32. (b) shows the construction of the data memory 2 in this embodiment. Numeral 511 denotes 
the inputted range image, 512 the computed normal vector, and 513 the polygonal patch model created 
according to this embodiment. 

The flow of overall processing in this embodiment will be described based upon the flowchart shown in 

75 Fig. 33. First, at step Si 31, a range image is inputted as by measuring an object by the laser range finder 7 
in accordance with the range-image input processing program 501, and the range image is stored in the 
data memory 2 as the range-image data 511. This input is not limited to one obtained by actual 
measurement, for an arrangement can be adopted in which data stored in an image data base are inputted. 
Next, at step SI 32, the inputted range-image data 511 (three-dimensional coordinate data of each point 

20 constituting the surface of the object) are processed in accordance with the normal-vector computation 
program 502. In order to obtain the normal vector, the inputted range image need only be spatially 
differentiated. This method of calculating a normal vector is described in the aforementioned paper entitled 
"A Hybrid Technique for Segmenting a Range Image based upon Differential Geometric Characteristics". 
In accordance with the polygonal patch model creating program 503, the polygonal patch model 513 is 

25 created at step S 133 by polygonal patches best suited for the surface shapes based upon the range-image 
data 51 1 and normal-vector data 512. 

A procedure for creating a polygonal patch model using a range image and a normal vector will be 
described with reference to the flowchart of Fig. 34. It will be assumed that the polygonal patches are 
quadrangular patches. 

30 First, at step Si 21, the range image is scanned sequentially, as shown in Fig. 35, to find a point (let this 
point be A in Fig. 36) that is on the object surface and not covered by a patch. Next, at step Si 22. a square 
ABCD having the found point A as one vertex and a side of length i (the densest spacing in the model to be 
created) is created. 

This is followed by step S123. at which it is determined whether three selected vertices B, C, D are all 
35 points still not covered by patches, and whether all of the internal products between a unit normal vector at 
point A and the unit normal vectors at points B. C. D are greater than a threshold value given in advance. If 
B. C, D are all points not yet covered and the internal products of the normal vectors are all greater than the 
threshold value, then the operation i = i + 1 is performed and step Si 22 is repeated. 

If any of the three vertices B, C, D is a point already covered by another patch or any of the internal 
40 products between the unit normal vector at point A and the unit normal vectors at points B, C. D is less than 
the threshold value given in advance, the vertices of the quadrangle are set at step Si 24, the three- 
dimensional coordinate values of the four set vertices A, B, C, D are read from the range image, the senses 
of the normal vectors are read from the normal-vector image, and the read data are registered as data 
indicative of the quadrangular patch. 
45 This is followed by step Si 25. at which it is determined whether scanning of the entire screen of the 
range image has ended. If there is a portion still unscanned, processing is repeated from step Si 21 onward. 
If scanning of the entire screen has ended, on the other hand, the program proceeds to step Si 26. Here a 
point on the side FG of another quadrangle occupied by any vertex constituting the above-described 
quadrangle is found in the registered quadrangular patch data. Then, at step Si 27. the three-dimensional 
50 coordinate values of the found vertex E are calculated from the equation of the straight line FG. and the 
quadrangular patch data are revised using these values as the coordinate values of the point E. When the 
revision of the three-dimensional coordinate values regarding all points such as point E end, the processing 
for creating the patch model is terminated. 

An example of part of the quadrangular patch data created by this procedure is illustrated in Fig. 38. 
In accordance with this embodiment, as described above, if, after representative points are selected 
densely from points of interest, a change in a normal vector between a point of interest and a selected point 
is less than a predetermined value, the representative points are re-selected at greater intervals. Therefore, 
a polygonal patch model can be created in such a manner that the spacing of representative points at 
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portions where change is great is narrowed and the spacing of representative points at smooth portions is 
widened. The shape of -he object is expressed with high precision using only a small amount of data. In 
addition, by describing the connection relationship between data points in the shape model, unnecessary 
data points can be eliminated or data points can be added later. 

(Example of editing of three-dimensional model 

In this embodiment, geometric data are generated from range-image data and an operation for 
transforming the geometric data is designated, thereby realizing an animation display by a simple operation. 

Fig. 39 illustrates the construction of the program memory 1 and data memory 2 according to the sixth 
embodiment, and Fig. 40 shows a flowchart of overall processing. 

In Fig. 39, (a) is a diagram showing the programs in the program memory 1 of the sixth embodiment. 
The programs include a range-image input processing program 601 the same as that in the fifth 
embodiment, a three-dimensional model constructing program 602 for generating various types of 3-D 
geometric data with regard to the inputted range image, a transformation data creating program 603 which 
designates/stores parameters for executing various transformation animations with regard to the 3-D 
geometric shape obtained, and a transformation animation display orogram 604 which, in accordance with 
the three-dimensional model and the animation parameters obtained, performs lighting and shading in 
dependence upon the operation for transforming the object, and carries out a transformation animation. 

In the data memory 2, numeral 61 1 denotes range-image data inputted from a range measuring sensor 
or range image DB, 612 the shape model constructed as 3-D geometric data by the processing of this 
embodiment, and 613 the object transformation parameters designated for animation by the user. 

The general features of processing according to this embodiment will be described with reference to 
the flowchart of Fig. 40. 

At step S181, the range image is inputted from a laser range finder or image data base, just as in the 
fourth and fifth embodiments. However, since central and cylindrical types of projections are available in 
addition to the parallel-projection system in order to measure a range image, in this step a transformation is 
made to the coordinate space of one projection system (the parallel-projection system) and subsequent 
processing is unified. A point at which range is greater than a fixed value is taken as infinity in order to be 
regarded as background; it is not regarded as part of an object. 

A three-dimensional shape model is created at step 5182. Though mesh, polygon and freely curved 
surfaces are available as the shape model, as mentioned above, it is desired that the typical shape be 
selected in dependence upon the characteristics of the object. That is, a polygon model is recommended 
for an artificial object having a large fiat surface, and a mesh or freely curved surface is suited to a smooth 
surface, such as the image of a face. An example of a polygon model is shown in Fig. 41. 

At step S183, light-source information (color, type of light source, attenuation factor), object attributes 
(object coior, various reflection coefficients, normal lines) are given for the geometric shape obtained, and 
the three-dimensional model is projected onto a two-dimensional plane to perform rendering by phong or 
glow shading. 

Since the image obtained here is only an image seen from one direction, the transformation mode for 
an animation operation is selected at step Si 84. Furthermore, the parameters for the transformation are 
designated at step Si 85. If it is determined at step Si 86 that all designations have ended, the program 
proceeds to step Si 87. If a plurality of transformation modes are merged, the program returns to step S184. 

The number of interpolations in a transformation is set at step Si 87. This processing is described 
below. An animation is created and displayed at step Si 88. 

In this embodiment, tapering, twisting and bending are the transformation modes. These will now be 
described. 

Fig. 42 illustrates the three exarroies of transformation, and Fig. 43A an example of transformation in 
which tapering and twisting are mergec. Fig. 43B shows an original color image corresponding to Fig. 43A. 

(1) Tapering [Fig. 42(a)] 

Though an example of a function for changing the x.y scale of an object along the z axis is illustrated, 
tapering can be achieved similarly also with regard to the other axes (x and y axes). This is expressed by 



X = r'x 
Y = r'y 
Z = 2 



(19) 
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and the scale factor is r = f(z). 

The transformation of an object can be simulated by changing the function f(z). Further, a normal vector 
is essential information in order to calculate shading when a three-dimensional object is subjected to 
5 rendering (shading). The direction of the normal also changes depending upon the transformation operation. 
A normal vector after transformation is as follows: 



10 
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(20) 



J5 

(2) Twisting [Fig. 42(c)] 

Here an example is illustrated in which the contour is rotated by an angle 0 (radians) in the x-y plane 
about the z axis. Twisting can be realized similarly with regard to the other axes (x and y axes) as well. 

20 

X = x*cos0 - y'sine 
Y = x"s\nd + y'cosd 
Z = z (21) 

25 For the angle of rotation, we have d = f(z). Here transformation of the object can be simulated by 
changing the function f(z). Similarly, the normal vector Nt after transformation is as follows: 
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(3) Bending [Fig. 42(d)] 

The bending function along the y axis is defined as set forth below, but the bending function is similar 
for the other axes (x and z axes) as well. 

. Here the curvature of bending is 1 k. the center of the bend is y = yo. the bending angle \s 6 = k(y'- 
yo), and bending is executed at yminSy^ymax. where 



50 



fymin if y< ytnin . 

y-<y if ymin < y < yiT 
[ymin if y ^ yaux 



(23) 



X = X 
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Y = 



-sine{z-l/ k)+yo'\'cosd{y-ymm) y < yr 
-sin0(2-l//:) + yo + cosfl(>'-ym«) y < y. 



ymia < y < ym«x 



70 



?5 



Vmin S V ^ Vi 

cos0(z-l/*) + l/)5: + sin0(y-ymin) y < ymin 
cos0(z-l/ik) + l/it + sin0(v->'max) y < y„,.x 

The normal vector N, after transformation is expressed as follows: 



20 



25 



(l-k:'z)Nt = 



1-k'z 

0 
0 



cos9 
sinG 



-sin9(l-kz' ) 
cos9(l-kz') 

... (24) 



where 



30 



k if ymiii ^ y < ytt 



35 



0 else 



The various transformation modes (tapering, twisting, bending) described above are designated by 
using a panel or the like in the window system. 
40 Thereafter, by opening another menu, the initial parameter of animation, the end parameter, the 
interpolation function and the number of interpolations every second are designated by the user. 

An example of a designation (relating to tapering) will be described with reference to Fig. 44. It is 
possible to designate the initial value (2201) and end value (2202) of the scale factor "r" by a slider or by a 
direct input of the numerical value, as illustrated in Fig. 44. When an input is made from one mode, the 
45 information is accurately reflected in the display of the other mode. 

There is a panel for designating, as another input, the number (2203) of generated frames of an image 
every second when an animation is performed; 1-30 images/sec is selected. 

When this done, a CPU interrupt is generated every designated interval and screen updating is 
executed on the side of the system at which animation is to take place. 
50 Creating an animation involves processing which places an extremely heavy burden upon the CPU. In a 
case where rendering is carried out solely by a CPU, a situation may arise in which it is difficult to achieve 
the above-mentioned performance of several frames per second. At such time, each frame is painted off- 
line and consecutive numbers are assigned to the frames. When the animation is viewed, these images are 
loaded at one time and it is possible to realize animation in which the impression is given of the images 
55 being turned over in rapid fashion. 

Finally, an interpolation function (2204) for deciding a parameter "r", which varies with time when 
animation is executed, is designated. This function indicates the manner in which a change will occur in the 
time direction from the inputted initial value and end value. The manner of change with time can be 
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designated by a variety of methods. 

The simplest is one in which the parameter value varies linearly in the time direction. However, when 

the user is not satisfied with this, it is necessary to designate a more complicated change. In Fig. 44, an 

interpolation function is designated utilizing a Bezier curve, by way of example. A function of this kind can 
5 be utilized as processing for curve generation in DTP. In general, a variation function of a parameter 

capable of being expressed in the form p = f(t) can be utilized as the interpolation function. 

In this embodiment, tapering, twisting and bending are used as the transformation operations. However, 

it is required that enlargement/reduction along the x, y, z axes, rotation about each axis and movement in 

the X, y, z directions also be realized as other transformations upon taking the transformation operations into 
w consideration. Processing for these cases is described in detail in Japanese translation of "Computer 

Graphics", by J. D. Foley and A. Van Dam. published by the Japan Computer Association, in 1984. pp. 253 

- 273. 

In shape modeling and animation operations, which are performed by designers in the prior art. a basic 
description can be created semi-automatically in accordance with this embodiment. As a result, the time 

15 involved in creating animations can be shortened and precision is improved. 

In accordance with the present invention, as described above, a three-dimensional model in which the 
shape of an object is represented correctly with a small amount of data can be created by selecting 
representative points that are closely spaced on complicated portions where shape is highly irregular, 
selecting representative points that are distantly spaced on large flat surfaces, and adopting these 

20 representative points as the vertices of polygonal patches. 

Further, the operation involved in creating an animation is simplified, the item required for creation is 
shorted and precision is improved. 

It should be noted that the present invention is not limited to a single apparatus but may be applied to a 
system composed of a plurality of apparatus so long as the functions of the present invention are 

25 implemented. It goes without, saying that the present invention can be applied also to a system in which the 
processing is carried out by supplying a program to the apparatus or system. 

As many apparently widely different embodiments of the present invention can be made without 
departing from the spirit and scope thereof, it is to be understood that the invention is not limited to the 
specific embodiments thereof except as defined in the appended claims. 

30 A three-dimensional model processing apparatus includes a three-dimensional data-format converter-for 
converting the data format of three-dimensional data inputted by a three-dimensional data input unit, a 
three-dimensional data processor for processing the three-dimensional data converted by the three- 
dimensional data-format converter, a three-dimensional model creating unit for creating a three-dimensional 
model from the three-dimensional data processed by the three-dimensional processor, and a three- 

35 dimensional model editor for editing the three-dimensional model created by the three-dimensional model 
creating unit and applying operations to the shape data. A three-dimensional model is created and edited 
irrespective of the type of three-dimensional data inputted in different modes. 

Claims 

40 

1. A three-dimensional model processing apparatus comprising: 

a three-dimensional data-format converter for converting data format of three-dimensional position 
data, indicative of an actual object, inputted by a three-dimensional data input unit; 

a three-dimensional data processor for processing the three-dimensional data converted by said 
45 three-dimensional data-format converter; 

a three-dimensional model creating unit for creating a three-dimensional model from the three- 
dimensional data processed by said three-dimensional processor; and 

a three-dimensional model editor for editing the three-dimensional model created by said three- 
dimensional model creating unit and applying operations to the shape data. 

50 

2. The apparatus according to claim 1, further comprising a three-dimensional model display unit for 
applying shading to the three-dimensional model edited by said three-dimensional model editor and 
displaying the resulting model on a display. 

55 3. The apparatus according to claim 1, further comprising a three-dimensional model reverse-converter for 
reverse-converting, into three-dimensional data having another format, the three-dimensional model 
edited by said three-dimensional model editor. 
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4. An image processing method comprising: 

a step of converting the type of an inputted range image and performing normalization along each 
axis; 

an arithmetic step of calculating a value of the converted range image and creating data which 
5 represents the properties of an object surface described by said range image; and 

a display step of displaying image information which represents the properties of said object 
surface based upon the data created at said arithmetic step. 

5. The method according to claim 4, wherein said arithmetic step includes a step of detecting a jump 
w edge from the range image. 

6. The method according to claim 4, wherein said arithmetic step includes a step of calculating a linear 
partial differential from the range image. 

75 7, The method according to claim 4, wherein said arithmetic step includes a step of calculating a 
quadratic partial differential from the range image. 

8. The method according to claim 4, wherein said arithmetic step includes a step of approximating the 
range image as a local linear plane and calculating coefficients of an equation thereof. 

20 

9. The method according to claim 4, wherein said arithmetic step includes a step of approximating the 
range image as a local quadratic surface and calculating coefficients of an equation thereof. 

10. The method according to claim 6, wherein said arithmetic step includes a step of calculating a 
25 quadratic partial differential from the range image, and a step of calculating Gaussian curvature using 

the calculated linear and quadratic partial differentials. 

11. The method according to claim 6, wherein said arithmetic step includes a step of calculating a 
quadratic partial differential from the range image, and a step of calculating mean curvature using the 

30 calculated linear and quadratic partial differentials. 

12. The method according to claim 6. wherein said arithmetic step includes a step of calculating a 
quadratic partial differential from the range image, and a step of calculating normal curvature using the 
calculated linear and quadratic partial differentials. 

35 

13. The method according to claim 6, wherein said arithmetic step includes a step of calculating a 
quadratic partial differential from the range image, and a step of calculating radius of curvature using 
the calculated linear and quadratic partial differentials. 

40 14. The method according to claim 10, wherein said arithmetic step includes a step of discriminating the 
sign of the calculated Gaussian curvature. 

15. The method according to claim 11, wherein said arithmetic step includes a step of discriminating the 
sign of the calculated mean curvature. 

45 

16. The method according to claim 4, wherein said arithmetic step includes a partial-differential calculating 
step of calculating linear and quadratic partial differentials from the range image, a curvature calculating 
step of calculating Gaussian curvature and mean curvature using the calculated linear and quadratic 
partial differentials, a sign discriminating step of discriminating the signs of the calculated Gaussian 

50 curvature and mean curvature, and a surface-category discriminating step of discriminating surface 
category by a combination of the discriminated sign of the Gaussian curvature and the discriminated 
sign of the mean curvature. 

17. The method according to claim 6, wherein said arithmetic step includes a step of calculating a unit 
55 normal vector using the calculated linear partial differential. 

18. The method according to claim 17, wherein said arithmetic step includes a step of detecting a roof 
edge from the calculated unit normal vector. 
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19. A method of creating a three-dimensional mode! from an inputted three-dimensional range image, 
comprising the steps of: 

inputting a three-dimensional range image obtained from a parallel projection in one direction; 
designating an axis of symmetry of an object of interest in three-dimensional space with respect to 
5 the given range image; and 

constructing a three-dimensional model by estimating the shape; of the object forming a back 
surface when viewed from said direction based upon the given range Image and the designated axis of 
symmetry. 

10 20. The method according to claim 19, wherein said designating step further includes steps of: 

projecting the given range image upon three two-dimensional planes and displaying a three-surface 
view; and 

designating the axis of symmetry of the object using the three-surface view. 

15 21. An apparatus for creating a three-dimensional model, comprising: 

holding means for holding an inputted three-dimensional range image; 

designating means for designating an axis of symmetry of an object of interest in three-dimensional 
space with respect to the given range image; and 

model forming means for constructing a three-dimensional model by estimating the shape of the 
20 object forming a back surface when viewed from said direction based upon the given range image and 
the designated axis of symmetry. 

22. The apparatus according to claim 21. further comprising means for storing the constructed three- 
dimensional model as an external-data file. 

25 

23. The apparatus according to claim 21, wherein said designating means designates the axis of symmetry 
of the object using a three-surface view. 

24. The apparatus according to claim 21, further comprising display means for subjecting the obtained 
30 three-dimensional model to processing such as lighting, shading and projection transformation, and 

displaying results of the processing in a two-dimensional plane. 

25. An image output method for outputting a three-dimensional shape as a two-dimensional Image, 
comprising the steps of extracting the sense of each point on a surface from given three-dimensional 

35 shape data, and outputting the extracted sense in an Identifiable manner. 

26. The method according to claim 25, wherein the output expresses the extracted sense in the form of 
direction and length of a line segment. 

40 27. The method according to claim 25, wherein the output expresses the extracted sense in the form of 
hue and saturation of color. 

28. The method according to claim 25, further comprising a step of having a user select outputted 
information and method of output from Information accompanying the extracted sense of a surface. 

45 

29. An Image output apparatus comprising: 

extracting means for extracting the sense of each point on a surface from given three-dimensional 
shape data; and 

output means for outputting the extracted sense In an identifiable manner. 

50 

30. The apparatus according to claim 29, wherein said output means expresses the extracted sense in the 
form of direction and length of a line segment. 

31. The apparatus according to claim 29, wherein said output means expresses the extracted sense in the 
55 form of hue and saturation of color. 

32. The apparatus according to claim 29, further comprising selecting means for allowing a user to select 
outputted information and method of output from information accompanying the extracted sense of a 
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surface. 

33. The apparatus according to claim 29. further comprising means for inputting and holding the three- 
dimensional shape data of an object. 

34. The apparatus according to claim 29. further comprising means for externally storing the extracted 
sense data of the surface. 

35. A range image processing method for obtaining, from an inputting range image, a group of polygons 
which cover an object surface represented by said range image, comprising: 

a surface-category classifying step for classifying, based upon the inputted range Image, the 
category of the surface constituting the object surface; and 

a polygonal patch model creating step for obtaining appropriate polygonal patches in conformity 
with the classified surface category, and creating a polygonal patch model of said object. 

36. The method according to claim 35, wherein polygonal patches having a small area are assigned to 
highly irregular surface categories among the classified categories. 

37. The method according to claim 35, wherein the smaller the irregularity of a surface category among the 
classified categories, the greater the spacing of representative points for selecting polygonal patches is 
made. 

38. A range image processing method for obtaining, from an inputting range image, a group of polygons 
which cover an object surface represented by said range image, comprising: 

a detecting step of detecting a point on the surface that is not covered by the group of polygons 
already obtained; 

a vertex-candidate selecting step of selecting, from other points on the surface, candidates for 
other vertices of a polygon one vertex of which is the point detected at said detecting step; 

a determining step of determining whether each point selected is covered by the group of polygons 
already obtained; and 

a comparing step which, in a case where all of the points are not covered by the group of 
polygons, is for comparing a normal direction at the detected point with a normal direction at each 
selected point; 

wherein when differences in direction are all less than a predetermined amount in each result of 
comparison performed at said comparing step, points constituting a polygon covering each selected 
point are selected anew, from other points on the surface, as candidates for other vertices of a polygon 
one vertex of which is the detected point. 

39. The method according to claim 38, wherein when a difference in direction is greater than a predeter- 
mined amount in any result of comparison performed at said comparing step, a polygon one vertex of 
which is the detected point and the other vertices of which are the selected points is added to said 
group of polygons. 

40. The method according to claim 38, wherein comparison of the normal directions is executed by 
comparing an internal product between normal vectors of two normal directions. 

41. The method according to claim 39, wherein comparison of the normal directions is executed by 
comparing an internal product between normal vectors of two normal directions. 

42. A range image processing method comprising; 

a model creating step of creating a three-dimensional model from an inputted range image; 

a transformation designating step of designating a type of transformation operation and parameters 
for transforming the three-dimensional model obtained at step model creating step; and 

an image generating step of generating an image, in which the three-dimensional model is 
successively transformed, based upon the designation made at said transformation designating step. 

43. The method according to claim 42. wherein a polygon model is used as said three-dimensional model. 
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44. The method according to claim 42, wherein the types of transformation operation include tapering, 
twisting and bending. 
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if(depth (p-1 . n) == background AND depth{p. n) == object) 



/* set binary 1. if depth{x. y) is object */ 
/* set binary 0, if depth(x. y) is background */ 

/* arrange flag according to depth (p, n) depth(p. n+1) depth(p+1. n) depth{p+1. 
/* regist(ind() ind() ind()) means set point index */ 
switch(flag) { 

case 1111;/* depth(p. n) depth{p, n+1) depth(fHl , n) depth(p+1 , n+1 ) * 

regist(ind(p, n) ind(p, n+1) ind(p+1» n)) 

regist(ind{p+1. n+1) ind(p+1, n) ind(p. n+1)) 

break; 
case 1110; 

regist(ind(p, n) ind(p, n+1) ind(p+1 , n)) 

break; 
case 1101; 

regist(ind(p. n) ind(p, n+1) ind(p+1. n+1)) 

break; 
case 1011; 

regist(ind(p, n) ind(p+1, n+1) ind(p+1, n)) 

break; 



caseoni; 

regist(ind{p+1, n+1) ind{p+1, n) ind(p, n+1)) 
break; 



case 1100; 
case 1010; 
case 1001; 
case 1000; 
caseOnO; 
case 0101; 
case 0100; 
case 0011; 
case 0010; 
case 0001 ; 
case 0000; 



regist depth(p. n) as CONTOUR; 



/ * step 502 */ 



do not regist; 
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polygon / * declaration of polygon data * / 

verlicds / * vertex Information follovws * / 

components=:x y z normal-x nonnal-y normal-z 

/ * attributes of each vertex 

always possesses three-dimensional 

coordinate values x, y, z 

if ft possesses normal vector, declare here * / 



numbers584 



( 78.00 
( 78.00 
(81.00 
(81.00 
(84.00 
(84.00 
( 87.00 



30.00 
33.00 
33.00 
30.00 
33.00 
30.00 
33.00 



-61.80 
-68.29 
-71.14 
-66.75 
-72.58 
-69.12 
-73.26 



(81.00 111.00 -67.23 

(84.00 111.00 -69.85 

(87.00 111.00 -72.17 

(90.00 111.00 -73.98 



/ * number of vertices contiguous to following * / 



-0.297 
-0.545 
-0.376 
-0.488 
-0.241 
-0.342 
-0.085 

-0.421 
-0.404 
0.199 
-0.007 



-0.954 
-0.733 
-0.739 
-0.740 
-0.711 
-0.733 
-0.681 



-0.031 ) 
-0.407 ) 
-0.559 ) 
-0.464 ) 
-0.661 ) 
-0.588 ) 
-0.727 ) 



/ * vertex number 1 * / 
/ * vertex numt)er 2*1 
I * vertex number 3 * / 
/ * */ 



0.767 -0.484 ) 

0.780 -0.477)/* */ 

0.979 -0.040 ) / * vertex number 583 * / 

1 .000 -0.022 ) / * vertex number 584 * / 



connectivity 

components = variable, i 
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14 3) 



15 
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5) 
7) 
9) 
19) 



(4 573 581 582 574) 

(4 574 582 583 575) 

(4 575 583 584 576) 
endpolygon 



/ * phase information follows * / 

/ * number of vertices constituting one 
polygon and series of vertex numbers 
thereof 



/ * end of polygon data * / 
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polyigon 
vertices 
components= 



number=8 




(-1.0 -1.0 


-1.0 


( 1.0 -1.0 


-1.0 


( 1.0 1.0 


-1.0 


(-1.0 1.0 


-1.0 


(-1.0 -1.0 


1.0 


( 1.0 -1.0 


1.0 


( 1.0 1.0 


1.0 


(-1.0 1.0 


1.0 


connectivity 





/ * declaration of polygon data * / 
/ * vertex infonmation * / 
X y z normal-x normal-y normal-z 

/ * vertex attributes, coordinate values 

+ normal Information * / 
/ * number of vertices * / 

0.57735 -0.57735 -0.57735) 
-0.57735 -0.57735 ) 
0.57735 



0.57735 
0.57735 
-0.57735 
-0.57735 
0.57735 
0.57735 
0.57735 



/ * vertex number 1 * / 
/ * vertex number 2 * / 



0.57735 ) 
0.57735 -0.57735 ) 
-0.57735 0.57735) 



-0.57735 
0.57735 
0.57735 



0.57735 ) 
0.57735 ) 
0.57735 ) 



components = variable, i 



(4 
(4 
(4 
(4 
(4 
(4 



2 
6 
5 
1 
6 
3 



3 
7 
8 
4 
2 
7 



4) 
3) 
7) 
8) 
1 ) 
8) 



/ * phase information * / 

/ * number of vertices contituting one polygon 
and series of vertex numbers thereof 



endpolygon 



/ * end of polygon data * / 
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G. 42 (a) 



G. 42 (b) 




TAPERING ALONG Z AXIS 



G. 42 (c) 



G. 42 (d) 




BENDING ALONG Y AXIS 
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3-D RENDERING OF ORIGINAL DATA 

F I G. 43A(a) 




RENDERING OF OBJECT SUBJECTED TO 
'iPERING AND TWISTING ALONG Z AXIS 
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V 



if(depth (p-1 , n) — background AND depth(p. n) <» object) 
regist d8pth(p, n) as CONTOUR; I 

if (depth (pi-l, n) == background AND d6pth(p, n) == object) 

regist depth(p. n) as CONTOUR; i / * step */ 

/* set binary 1, if depth(x. y) is object */ j 
/* set binary 0. if depth(x. y) is background */ 

/* arrange flag according to depth (p. n) depth(p. n+x) depth(p+x. n) depth(p+x, n+x) order 
/* regist(indO ind() ind()) means set point inde^*/ 
switch(flag) { 

case 1111;/* deptti(p, n) depth(p, n>x) depth(p+x, n) depth(p+x, n+x) */ 
regist(ind(p. n) ind(p. n+x) ind{p+x, n)) 
regist(ind(p+x, n+x) ind(p+x. n) ind(p, n+x)) 
break; 
case 1110; 

regist(ind(p. n) ind(p, n+x) ind(p+x, n)) 
break; 
case 1101; 

regist(ind(p, n) ind(p, n+x) ind(p+x, n+x)) 
break; 
case 1011; 

regist(ind(p, n) ind(p+x, n+x) ind(p+x, n)) 
break; 
case 0111; 

regist{ind(p+x, n+x) ind(p+x, n) ind(p, n+x)) 
break; 



case 1100; 
case 1010 
case 1001 
case 1000 
case 01 10 
case 0101: 
case 0100; 
case 001 1 
case 0010 
case 0001 
case 0000 



do not regist; 
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